<html>
<head><meta charset="utf-8"><title>non-temporal stores for `mm_profdata` · t-compiler/performance · Zulip Chat Archive</title></head>
<h2>Stream: <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/index.html">t-compiler/performance</a></h2>
<h3>Topic: <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/non-temporal.20stores.20for.20.60mm_profdata.60.html">non-temporal stores for `mm_profdata`</a></h3>

<hr>

<base href="https://rust-lang.zulipchat.com">

<head><link href="https://rust-lang.github.io/zulip_archive/style.css" rel="stylesheet"></head>

<a name="215497210"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/non-temporal%20stores%20for%20%60mm_profdata%60/near/215497210" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/non-temporal.20stores.20for.20.60mm_profdata.60.html#215497210">(Nov 03 2020 at 19:03)</a>:</h4>
<p><span class="user-mention" data-user-id="125250">@Wesley Wiser</span> <span class="user-mention" data-user-id="124287">@mw</span> wild random idea for <code>measureme</code>: non-temporal stores to memory for events and maybe even tweaking the paging system so the event stream itself doesn't need buffering. specifically I'm imaginging we would trash the cache less this way, but I haven't actually measured what the overhead of <code>-Z self-profile</code> is, compared to not having it enabled</p>



<a name="215504209"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/non-temporal%20stores%20for%20%60mm_profdata%60/near/215504209" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Vadim Petrochenkov <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/non-temporal.20stores.20for.20.60mm_profdata.60.html#215504209">(Nov 03 2020 at 20:09)</a>:</h4>
<p><span class="user-mention" data-user-id="119009">@eddyb</span> <br>
Have you measured how much faster does the profiling works if the profile is written into memory rather than into a file?<br>
Non-temporal stores are not a wild idea, streaming data to somewhere is exactly why they exist, but they are unlikely to help if you have <em>too</em> much data so that it's IO bound (or even if there's not so much data, but still enough for the profiling to be store port bound).</p>



<a name="215504287"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/non-temporal%20stores%20for%20%60mm_profdata%60/near/215504287" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/non-temporal.20stores.20for.20.60mm_profdata.60.html#215504287">(Nov 03 2020 at 20:10)</a>:</h4>
<p>I think the implementation has  changed a bunch, but I've never measured the total overhead</p>



<a name="215504308"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/non-temporal%20stores%20for%20%60mm_profdata%60/near/215504308" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/non-temporal.20stores.20for.20.60mm_profdata.60.html#215504308">(Nov 03 2020 at 20:10)</a>:</h4>
<p>I didn't even realized the <code>mmap</code> I saw was an anonymous one etc.</p>



<a name="215504316"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/non-temporal%20stores%20for%20%60mm_profdata%60/near/215504316" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/non-temporal.20stores.20for.20.60mm_profdata.60.html#215504316">(Nov 03 2020 at 20:10)</a>:</h4>
<p>wait let me split this out into its own topic</p>



<a name="215712472"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/non-temporal%20stores%20for%20%60mm_profdata%60/near/215712472" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/non-temporal.20stores.20for.20.60mm_profdata.60.html#215712472">(Nov 05 2020 at 13:44)</a>:</h4>
<p>the implementation has changed quite a bit recently. we now write everything into a smallish in-memory buffer and then, once that is full, write that to disk.</p>



<a name="215712633"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/non-temporal%20stores%20for%20%60mm_profdata%60/near/215712633" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/non-temporal.20stores.20for.20.60mm_profdata.60.html#215712633">(Nov 05 2020 at 13:45)</a>:</h4>
<p>access to the buffer is synchronized via a mutex and can happen from multiple threads. I'm not sure how that would affect non-temporal stores.</p>



<a name="215712784"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/non-temporal%20stores%20for%20%60mm_profdata%60/near/215712784" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/non-temporal.20stores.20for.20.60mm_profdata.60.html#215712784">(Nov 05 2020 at 13:46)</a>:</h4>
<p>In theory it sounds like a valid use case for them though.</p>



<a name="215713046"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/non-temporal%20stores%20for%20%60mm_profdata%60/near/215713046" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/non-temporal.20stores.20for.20.60mm_profdata.60.html#215713046">(Nov 05 2020 at 13:48)</a>:</h4>
<p>what I found while doing the current implementation: the code is not really I/O bound (at least on Linux) because writing to disk without flushing will actually just copy the data into kernel memory and then return, while disk I/O happens in the background.</p>



<a name="215713108"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/non-temporal%20stores%20for%20%60mm_profdata%60/near/215713108" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/non-temporal.20stores.20for.20.60mm_profdata.60.html#215713108">(Nov 05 2020 at 13:49)</a>:</h4>
<p>which is why having a manual implementation of writing stuff to disk in a background thread made almost no difference</p>



<a name="215713309"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/non-temporal%20stores%20for%20%60mm_profdata%60/near/215713309" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/non-temporal.20stores.20for.20.60mm_profdata.60.html#215713309">(Nov 05 2020 at 13:50)</a>:</h4>
<p>I'm sure there is still a lot of potential for making this faster (esp. if using platform-specific stuff like IO_uring) but I ran out of time and was satisfied with the new implementation performing as well as the previous one.</p>



<a name="215713429"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/non-temporal%20stores%20for%20%60mm_profdata%60/near/215713429" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/non-temporal.20stores.20for.20.60mm_profdata.60.html#215713429">(Nov 05 2020 at 13:51)</a>:</h4>
<p>as far as I could tell, the current implementation's bottleneck is the mutex around the memory buffer</p>



<a name="215713545"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/non-temporal%20stores%20for%20%60mm_profdata%60/near/215713545" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/non-temporal.20stores.20for.20.60mm_profdata.60.html#215713545">(Nov 05 2020 at 13:52)</a>:</h4>
<p><span class="user-mention" data-user-id="125250">@Wesley Wiser</span> mentioned wanting to try using thread-local storage to git rid of that</p>



<a name="215713737"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/non-temporal%20stores%20for%20%60mm_profdata%60/near/215713737" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/non-temporal.20stores.20for.20.60mm_profdata.60.html#215713737">(Nov 05 2020 at 13:53)</a>:</h4>
<p>that should not be too hard, if we can assume that recording code does not run on the main thread (which I think it doesn't for rustc, right?)</p>



<a name="215750377"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/non-temporal%20stores%20for%20%60mm_profdata%60/near/215750377" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/non-temporal.20stores.20for.20.60mm_profdata.60.html#215750377">(Nov 05 2020 at 17:58)</a>:</h4>
<p><span class="user-mention" data-user-id="124287">@mw</span> you can avoid the mutex if you don't page events :P</p>



<a name="215750526"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/non-temporal%20stores%20for%20%60mm_profdata%60/near/215750526" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/non-temporal.20stores.20for.20.60mm_profdata.60.html#215750526">(Nov 05 2020 at 17:59)</a>:</h4>
<p>I've never seen the point of "background threads" tbh, and I want to eventually take a look at any we might still be using</p>



<a name="215750781"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/non-temporal%20stores%20for%20%60mm_profdata%60/near/215750781" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/non-temporal.20stores.20for.20.60mm_profdata.60.html#215750781">(Nov 05 2020 at 18:00)</a>:</h4>
<p>with <code>mmap</code>'d files, you avoid hitting that userspace-&gt;kernel copy from <code>write</code>, IIUC, and non-temporal stores <em>to that</em> could be interesting in that only the kernel will bring them in cache (if at all, it might just send them to DMA and they might never show up in cache until e.g. <code>summarize</code> reads the file)</p>



<a name="215771531"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/non-temporal%20stores%20for%20%60mm_profdata%60/near/215771531" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Vadim Petrochenkov <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/non-temporal.20stores.20for.20.60mm_profdata.60.html#215771531">(Nov 05 2020 at 20:45)</a>:</h4>
<p>Also, you can put a guard page after the mmaped page, and handle all the necessary unmapping/remapping in the signal handler firing on store to that guard page.<br>
Then you get the purest stream of stores not tainted by any bound checking.<br>
I'm not sure it's a reasonable thing to do in this case though (but in some cases it is).</p>



<a name="215820290"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/non-temporal%20stores%20for%20%60mm_profdata%60/near/215820290" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/non-temporal.20stores.20for.20.60mm_profdata.60.html#215820290">(Nov 06 2020 at 09:11)</a>:</h4>
<p><span class="user-mention silent" data-user-id="119009">eddyb</span> <a href="#narrow/stream/247081-t-compiler.2Fperformance/topic/non-temporal.20stores.20for.20.60mm_profdata.60/near/215750377">said</a>:</p>
<blockquote>
<p><span class="user-mention silent" data-user-id="124287">mw</span> you can avoid the mutex if you don't page events :P</p>
</blockquote>
<p>Well, we moved away from having one file per stream and towards paging things because people kept complaining that they want to have everything in a single file <code>:)</code></p>



<a name="215820563"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/non-temporal%20stores%20for%20%60mm_profdata%60/near/215820563" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/non-temporal.20stores.20for.20.60mm_profdata.60.html#215820563">(Nov 06 2020 at 09:14)</a>:</h4>
<p>unfortunately both the string table and the events stream both can be gigabytes in size, so we kinda have to write everything to disk asap if we want avoid both high memory consumption and dumping lots of data to disk at once, right?</p>



<a name="215820927"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/non-temporal%20stores%20for%20%60mm_profdata%60/near/215820927" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/non-temporal.20stores.20for.20.60mm_profdata.60.html#215820927">(Nov 06 2020 at 09:18)</a>:</h4>
<p>it does sound like a lot of fun to optimize this whole thing further, if one can spend a couple of months on it <code>:)</code> From a practical standpoint though, I think it would be good to first proof that the current overhead is high enough to warrant the additional complexity. The current implementation is rather simple.</p>



<a name="215821063"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/non-temporal%20stores%20for%20%60mm_profdata%60/near/215821063" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/non-temporal.20stores.20for.20.60mm_profdata.60.html#215821063">(Nov 06 2020 at 09:19)</a>:</h4>
<p>that being said, I of course have no objections to do simple optimizations that keep things as maintainable as they are <code>:)</code></p>



<a name="215830574"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/non-temporal%20stores%20for%20%60mm_profdata%60/near/215830574" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/non-temporal.20stores.20for.20.60mm_profdata.60.html#215830574">(Nov 06 2020 at 10:58)</a>:</h4>
<p><span class="user-mention" data-user-id="124287">@mw</span> yeah, I was one of those people :P. but my point is that you can page everything else while keeping individual events basically their own "page" (you only need like one bit to indicate you have an event and not a page. ooor you could always leave a "footer" at the end of each page that gets filled by the number of bytes from it to the next page, when the next page gets written, that way iterating the pages shouldn't be slower than today)</p>



<a name="215830645"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/non-temporal%20stores%20for%20%60mm_profdata%60/near/215830645" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/non-temporal.20stores.20for.20.60mm_profdata.60.html#215830645">(Nov 06 2020 at 10:59)</a>:</h4>
<p>going forward it might be nice to keep compatibility in the data format even if not in the API</p>



<a name="215830766"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/non-temporal%20stores%20for%20%60mm_profdata%60/near/215830766" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/non-temporal.20stores.20for.20.60mm_profdata.60.html#215830766">(Nov 06 2020 at 11:00)</a>:</h4>
<p>at least we could argue that multiple files is too much to keep around but in single-file we could have several approaches, and "just" paper over the differences between them when decoding</p>



<a name="215830916"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/non-temporal%20stores%20for%20%60mm_profdata%60/near/215830916" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/non-temporal.20stores.20for.20.60mm_profdata.60.html#215830916">(Nov 06 2020 at 11:02)</a>:</h4>
<p>actually... hmm... if the pages are flexible enough, we don't need to change the format, haha</p>



<a name="215830957"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/non-temporal%20stores%20for%20%60mm_profdata%60/near/215830957" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/non-temporal.20stores.20for.20.60mm_profdata.60.html#215830957">(Nov 06 2020 at 11:02)</a>:</h4>
<p>just always start an "events" page after any other page, and fill in the details when writing in the next page</p>



<a name="215831042"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/non-temporal%20stores%20for%20%60mm_profdata%60/near/215831042" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/non-temporal.20stores.20for.20.60mm_profdata.60.html#215831042">(Nov 06 2020 at 11:03)</a>:</h4>
<p>maybe I should open an issue</p>



<a name="215831127"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/non-temporal%20stores%20for%20%60mm_profdata%60/near/215831127" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/non-temporal.20stores.20for.20.60mm_profdata.60.html#215831127">(Nov 06 2020 at 11:04)</a>:</h4>
<p>oh yeaaaah the format is great :D</p>



<a name="215831809"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/non-temporal%20stores%20for%20%60mm_profdata%60/near/215831809" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/non-temporal.20stores.20for.20.60mm_profdata.60.html#215831809">(Nov 06 2020 at 11:10)</a>:</h4>
<p>although... there is that whole thing where I don't think we ever properly <code>mmap</code>'d files?? what's up with that</p>



<a name="215833829"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/non-temporal%20stores%20for%20%60mm_profdata%60/near/215833829" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/non-temporal.20stores.20for.20.60mm_profdata.60.html#215833829">(Nov 06 2020 at 11:33)</a>:</h4>
<p>done <a href="https://github.com/rust-lang/measureme/issues/144">https://github.com/rust-lang/measureme/issues/144</a></p>



<a name="216205532"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/non-temporal%20stores%20for%20%60mm_profdata%60/near/216205532" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/non-temporal.20stores.20for.20.60mm_profdata.60.html#216205532">(Nov 10 2020 at 12:43)</a>:</h4>
<p>I had an implementation with <code>mmap</code>'d files a while ago here: <a href="https://github.com/rust-lang/measureme/pull/16">https://github.com/rust-lang/measureme/pull/16</a><br>
I abandoned it after it wasn't faster than the other <code>mmap</code> based implementation. It always made me uneasy to allocate a number of huge files on disk (3x 1 GB in that case) and only truncating them in the <code>Drop</code> impl of the <code>SerializationSink</code>.</p>



<a name="216205722"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/non-temporal%20stores%20for%20%60mm_profdata%60/near/216205722" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/non-temporal.20stores.20for.20.60mm_profdata.60.html#216205722">(Nov 10 2020 at 12:45)</a>:</h4>
<p>I'm also sure that I did not eke out all the performance to be had there. I just stopped looking into it because I ran out of time.</p>



<a name="216451513"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/non-temporal%20stores%20for%20%60mm_profdata%60/near/216451513" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/non-temporal.20stores.20for.20.60mm_profdata.60.html#216451513">(Nov 12 2020 at 10:38)</a>:</h4>
<p><span class="user-mention" data-user-id="124287">@mw</span> ah, good to know! ideally we could inform the OS to lazily allocate space on disk, but I'm not sure how to do that exactly</p>



<a name="216451777"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/non-temporal%20stores%20for%20%60mm_profdata%60/near/216451777" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> bjorn3 <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/non-temporal.20stores.20for.20.60mm_profdata.60.html#216451777">(Nov 12 2020 at 10:40)</a>:</h4>
<p>On the terminal you can use <code>fallocate --posix --length 1G</code>/<code>truncate -s 1g</code> to create a sparse file. I don't know if Windows supports it though.</p>



<a name="216452236"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/non-temporal%20stores%20for%20%60mm_profdata%60/near/216452236" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/non-temporal.20stores.20for.20.60mm_profdata.60.html#216452236">(Nov 12 2020 at 10:45)</a>:</h4>
<p>interesting, I guess there's no way to have a <code>mmap</code> where writes determine the size of the file</p>



<a name="216452540"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/non-temporal%20stores%20for%20%60mm_profdata%60/near/216452540" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/non-temporal.20stores.20for.20.60mm_profdata.60.html#216452540">(Nov 12 2020 at 10:49)</a>:</h4>
<p><span class="user-mention" data-user-id="124287">@mw</span> there is one hilarious way to deal with having a tiny <code>mmap</code> size and running out of it, without locks in the fast path: put the fallback in the bounds checking failure path. you can probably even remap the same area of memory but pointing to the next same-size chunk on disk etc.</p>



<a name="216452767"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/non-temporal%20stores%20for%20%60mm_profdata%60/near/216452767" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/non-temporal.20stores.20for.20.60mm_profdata.60.html#216452767">(Nov 12 2020 at 10:51)</a>:</h4>
<p>you just need that chunk size to be small enough relative to <code>AtomicUsize</code> to allow "going on" by the number of maximum threads you might have, and not overflow (or I guess you can panic on overflow instead of panicking on out-of-bounds, even if the data on disk might already be corrupted by the time everything winds down)</p>



<a name="216454826"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/non-temporal%20stores%20for%20%60mm_profdata%60/near/216454826" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/non-temporal.20stores.20for.20.60mm_profdata.60.html#216454826">(Nov 12 2020 at 11:13)</a>:</h4>
<p><span class="user-mention" data-user-id="124287">@mw</span> wrote it down here <a href="https://github.com/rust-lang/measureme/issues/144#issuecomment-726012677">https://github.com/rust-lang/measureme/issues/144#issuecomment-726012677</a></p>



<a name="216454847"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/non-temporal%20stores%20for%20%60mm_profdata%60/near/216454847" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/non-temporal.20stores.20for.20.60mm_profdata.60.html#216454847">(Nov 12 2020 at 11:13)</a>:</h4>
<p>starting to wonder if we're seriously mishandling threads <span aria-label="sweat smile" class="emoji emoji-1f605" role="img" title="sweat smile">:sweat_smile:</span></p>



<a name="216454902"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/non-temporal%20stores%20for%20%60mm_profdata%60/near/216454902" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/non-temporal.20stores.20for.20.60mm_profdata.60.html#216454902">(Nov 12 2020 at 11:14)</a>:</h4>
<p>maybe each should have their own file? wouldn't work great for parallel rustc</p>



<a name="216462926"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/non-temporal%20stores%20for%20%60mm_profdata%60/near/216462926" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Diggsey <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/non-temporal.20stores.20for.20.60mm_profdata.60.html#216462926">(Nov 12 2020 at 12:51)</a>:</h4>
<p>I'm probably not aware of all the design constraints here, but wouldn't it be simpler to have a dedicated thread writing to the file? Idea being that you'd have a thread-local ring-buffer that all the normal threads write to. The background thread then has the job of gathering from those buffers and writing out to a file.</p>



<a name="216488218"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/non-temporal%20stores%20for%20%60mm_profdata%60/near/216488218" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/non-temporal.20stores.20for.20.60mm_profdata.60.html#216488218">(Nov 12 2020 at 16:02)</a>:</h4>
<p>I'm still not sure what problem such threads solve in userspace</p>



<a name="216488235"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/non-temporal%20stores%20for%20%60mm_profdata%60/near/216488235" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/non-temporal.20stores.20for.20.60mm_profdata.60.html#216488235">(Nov 12 2020 at 16:02)</a>:</h4>
<p>since you're effectively reimplementing part of the OS</p>



<a name="216488261"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/non-temporal%20stores%20for%20%60mm_profdata%60/near/216488261" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/non-temporal.20stores.20for.20.60mm_profdata.60.html#216488261">(Nov 12 2020 at 16:02)</a>:</h4>
<p>I guess batching syscalls?</p>



<a name="216488346"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/non-temporal%20stores%20for%20%60mm_profdata%60/near/216488346" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/non-temporal.20stores.20for.20.60mm_profdata.60.html#216488346">(Nov 12 2020 at 16:03)</a>:</h4>
<p>you can still run out of space in the buffer so it's not easier than what I'm suggesting, nor more deterministic</p>



<a name="216544385"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/non-temporal%20stores%20for%20%60mm_profdata%60/near/216544385" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Diggsey <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/non-temporal.20stores.20for.20.60mm_profdata.60.html#216544385">(Nov 12 2020 at 23:29)</a>:</h4>
<p>Well you don't need any locking when accessing the file because only one thread does that. Passing buffers back and forth in a lock-free fashion is a solved problem. Each thread could have two buffers, and each time one is filled it passes it to the writer thread to flush to disk, and switches to the other buffer. When the writer is done it sends the original buffer back.</p>



<a name="216570535"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/non-temporal%20stores%20for%20%60mm_profdata%60/near/216570535" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/non-temporal.20stores.20for.20.60mm_profdata.60.html#216570535">(Nov 13 2020 at 05:55)</a>:</h4>
<p>ah hmm double-buffering</p>



<a name="216570616"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/non-temporal%20stores%20for%20%60mm_profdata%60/near/216570616" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/non-temporal.20stores.20for.20.60mm_profdata.60.html#216570616">(Nov 13 2020 at 05:56)</a>:</h4>
<p><span class="user-mention" data-user-id="295632">@Diggory Blake</span> but how would the threads get interleaved? if you don't interleave at the event level, you can also do this without a separate thread</p>



<a name="216570666"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/non-temporal%20stores%20for%20%60mm_profdata%60/near/216570666" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/non-temporal.20stores.20for.20.60mm_profdata.60.html#216570666">(Nov 13 2020 at 05:57)</a>:</h4>
<p>like if the result on disk is significantly different, you can also tweak my scheme to do something similar (and presumably simplify it because of that)</p>



<a name="216570719"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/non-temporal%20stores%20for%20%60mm_profdata%60/near/216570719" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/non-temporal.20stores.20for.20.60mm_profdata.60.html#216570719">(Nov 13 2020 at 05:58)</a>:</h4>
<p>anyway, the advantage of <code>mmap</code>d IMO files is removing one set of copies (and a bunch of userspace work)</p>



<a name="216590317"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/non-temporal%20stores%20for%20%60mm_profdata%60/near/216590317" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/non-temporal.20stores.20for.20.60mm_profdata.60.html#216590317">(Nov 13 2020 at 10:01)</a>:</h4>
<p><span class="user-mention" data-user-id="125250">@Wesley Wiser</span> mentioned that he wanted to look into using TLS -- which I think is the only way to have per-thread buffers (due to <code>Profiler</code> having to be <code>Sync</code>, IIUC)</p>



<hr><p>Last updated: Aug 07 2021 at 22:04 UTC</p>
</html>